Reflections on Information Retrieval Evaluation

Information retrieval (IR) research has primarily consisted of two paradigms: systemsoriented research and user studies. Systems-oriented research includes IR algorithm development and evaluation and, to some degree, human-system interaction. User studies include human information behavior and information seeking research. These paradigms have contributed new IR algorithms and insights into IR processes. However, problematic issues concerning IR evaluation exist in both communities. For example, research methods, measures, and metrics that can yield generalizable results are problematic for systemsoriented research and user studies. We propose that establishing a dialog between these communities and synthesizing their methods may increase the effectiveness of both. In this paper, a brief overview of IR evaluation, including systems-oriented research and user studies, is presented and problematic issues in IR evaluation are discussed. We synthesize these approaches and propose a framework for evaluation that is based on attributes that have been shown to influence adoption of innovations (Rogers, 1995).